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(57) ABSTRACT 

A medical imaging display system includes a memory (40) 
for storing first image data representative of a region of 
interest. The memory (40) stores image data generated by 
medical imaging devices such as magnetic resonance 
devices (20), computed tomography devices (22), nuclear 
imaging devices (26,28,30), and ultrasound devices. 
Typically, image data from these devices is obtained some 
time prior to a surgical event and users may access this data 
in planning for the surgical event. A processor (42), in data 
communication with the memory, is organized under a 
component object modeling architecture. The processor (42) 
is connected to a user interface (10) for providing user 
requests to the processor. Thus, in response to user action via 
the user interface (10), the processor (42) determines an 
object (54) adapted to act on the request, selects a handle 
(60) for the determined object and, employs the object via 
the handle to act on the request. Additionally, the image 
guided surgical system also includes a source (48) of sub- 
stantially real time image data generated in the surgical 
theater, such as spectroscopy devices, which can also be 
manipulated via software objects to display desired portions 
of the region of interest Any of these images can be viewed 
on a display (46) in a planning environment, in the surgical 
suite, or even by a consultant, geographically remote from 
the surgical site. 

23 Claims, 5 Drawing Sheets 
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METHOD AND APPARATUS FOR MEDICAL 

IMAGE DISPLAY FOR SURGICAL TOOL 
PLANNING AND NAVIGATION IN CLINICAL 
ENVIRONMENTS 

5 

BACKGROUND OF THE INVENTION 

The present invention relates to the medical image display 
art. It finds particular application in connection with medical 
image displays in the surgical suite, and in the surgical 10 
planning stages where a need exists to quickly display large 
image files to enable an understanding of the anatomy of a 
patient, and thus will be described with particular reference 
thereto. It is to be appreciated however, that the invention 
will also find application in conjunction with other types of 15 
imaging where data on remote sensors is made available to 
a client via a component object model architecture. 

Increasing amounts of information are available to medi- 
cal personnel in the course of planning procedures and even 2Q 
during actual surgical events. Continued advances in medi- 
cal imaging systems such as magnetic resonance imagers, 
computer tomography devices, nuclear imaging equipment, 
positron emission tomography and ultrasound often provide 
a variety of 2-D and 3-D image files of increasing size and ^ 
optimized for imaging of specific environments. Often sur- 
gical planning requires reference to images obtained from 
one or more of the above listed imagers, and still other 
images are needed during the surgical procedure. 
Accordingly, image displays must be able to quickly access 3Q 
and manipulate image files. However, current display tech- 
niques have limitations due to large volumes of data asso- 
ciated with large imaged volumes, high resolution images, 
multiple images of a com mon volum e from different imag- 
ing modalities, multiple temporally offset images of a com- 35 
mon volujnef (Hn6), the number of colors used, and other 
factors which make it difficult to optimize methods of 
accessing data from the memories of each of the above listed 
imagers. Moreover, utilizing the volumetric data of different 
imaging modalities is complex due to the different 
resolutions, different coordinates, different orthogonal and 
oblique planes, and the like. 

surgical planning also requires deeper cut sections of the 
3-D organs in multiple views to understand the anatomy of 
these organs. Typically, a series of planes across each 45 
proposed surgical path are indexed sequentially to under- 
stand the significance of a proposed path. The selected cut 
sections, slices, 3D renderings, and other views are typically 
needed rapidly in the clinical environment, particularly 
when a surgical procedure is in progress. This heavy burden 50 
on computer memory reduces display speed. The control of 
the data access diminishes as time progresses. Worse yet, the 
heavy data access and movement burden sometimes causes 
the computers to freeze removing the diagnostic tools from 
the physician conducting the planning process or the surgi- 55 
cal procedure. 

The present invention contemplates a new, improved 
method and apparatus for diagnostic image- processing 
graphical^nd display which overcomes the above - 
referenced mmculties and others. 60 

SUMMARY OF THE INVENTION 

In accordance with an embodiment of the present 
invention, a method of medical imaging includes collecting 
medical image data, representative bit a region oL'mterest. 65 
Responsive to receipt of a request for manipulation of the~ v 
image data, a software process is determined to at least 
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partially accomplish the request. A handle for the deter- 
mined process is selected, and the request is processed via 
the handle. 

In accordance with another embodiment of the present 
invention, the method further includes displaying the pro- 
cessed image data. 

In accordance with another embodiment of the present 
invention, the collecting image data step includes prior to a 
surgical event, receiving first image data from at least one 
medical imaging device. The medical imaging device is 
selected from the set of a magnetic resonance device, a 
computed tomography device, a nuclear imaging device, a 
Positron emission tomography device, and an ultrasound 
device. The first image data is then stored in a memory. 

In accordance with another aspect of the present 
invention, the collecting image data step further includes 
during the surgical event receiving second image data, and 
storing the second image data in a memory. 

In accordance with another aspect of the present 
invention, the request for manipulation of the image data 
includes an input from a user interface. The determining 
process step includes referencing a registry to detect the 
registration of a process adapted to accomplish the request. 
From the registry, a handle is noted associated with the 
process. 

In accordance with another aspect of the present 
invention, the process resides in a component object mod- 
eling architecture and is adapted to make requests of other 
processes. The processing request step includes processing 
the request, and at determined locations in the process, 
formatting another request for another process to further 
accomplish the original request. 

In accordance with another aspect of the present 
invention, the method further includes planning the surgical 
event by performing tasks, including manipulating the user 
interface to selectively control image data displayed. 

In accordance with another aspect of the present 
invention, the method further includes performing the sur- 
gical event by performing tasks including manipulating the 
user interface to selectively control displays of the first and 
second image data. 

In accordance with another embodiment of the present 
invention, a medical imaging system includes an imaging 
device for producing image data representative of a region 
of interest. A memory stores the image data and the proces- 
sor manipulates at least portions of the image data for 
viewing on a display. A method of controlling the process 
includes the computer implemented steps of determining a 
process adapted to act on the request in response to a 
manipulation request from a client. A handle is determined 
which is associated with a component comprising the deter- 
mined process, and the request is forwarded to the deter- 
mined process via the handle. 

In accordance with another aspect of the present 
invention, the determining a process step includes returning 
an identifier from a registry of identifiers. The returned 
identifier is associated with the component including the 
process. 

In accordance with another embodiment of the present 
invention, a medical imaging display system includes a 
memory storing first image data representative of a region of 
interest. A processor in data communication with a memory 
is additionally connected to a user interface for manipulating 
portions of image data in response to requests. The processor 
is controlled by component object modeled software to (a) 
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determine an object adapted to act on the request, (b) select computer mouse, for example. The COM-architecture layer 

a handle for the determined object and, (c) employ the object B is the main software layer where the core applications are 

via the handle to act on the request. written in an object oriented Language such as C++. This 

In accordance with another aspect of the present ^ fa ' re * erred L to u as me software layout layer, since it 
invention, the imaging display system further includes a 3 » the foundation of the ^tem-Those dulled in the art 

A . . *• i » j ; T »u u- u 1 recognize that constructing the COM-architecture layer B 

oisplay operatively connected to the processor which selec- {q ^ QQM ^ udbhs ^ caablcs y C0M 

tively displays the mampulated image data. based concepts to be employed, for example in manipulaung 
In accordance with another aspect of the present image data. These concepts are threading, category manager, 
invention, the display system includes an image guided componen t category, containment, aggregation, 
surgery planning system in which an operator selects views encapsulation, inheritance, polymorphism, over-riding, class 
from the image data to plan a surgical event. factory, apartment and free threading, marshaling of data, 
In accordance with another aspect of the present and the like. Moreover, artisans will appreciate that a 
invention, the system further includes a source of substan- COM-architecture layer so constructed will have improved 
tially real time image data representative of the region of 1$ display performance and increased stability as will be more 
interest. A second object, selectively accessible to the fully discussed below. The server layer C, contains system 
processor, is adapted to manipulate the real time image data. and image data information. Typically, medical image data 
In accordance with another aspect of the present is stored in this layer in D1COM format. This layer also 
invention, the display system includes an image guided desirably enables various connections to other platforms, 
surgery system in which an operator selects views from the 20 Reference to FIG. 2, illustrates several such other plat- 
image data and real time image data to display during a forms connectable to server layer C. These platforms include 
surgical event. a magnetic resonance imaging (MRI) system 20, a comput- 
One advantage of the present invention resides in the prized tomography (CI) scanner 22, a digital x-ray radio- 
utilization of component object modeling architecture to graphic or fluorographic system 24, a positron emission 
improve the speed of display updates and appearance of the 25 tomography (PET^scanner, 26, a single photon emission 
display during manipulation. computed tomography (SPECT) scanner 28, and a nuclear 
Another advantage of the present invention resides in the ^ m **• However, those skilled in the art will 
ability to perform remote surgery from a remote server. appreciate other known and yet to be developed imaging 
A . , 0 , , modalities can be employed analogously. That is, any image 
Another advantage of the present resides ;in the systems data atioa equipment can be ^ed without loss of 
flexibility to include other software, or third party software m me prescnt mvcntion . Typify, images are 
products within a COM^ased architecture system. collected one at a time from one, two, or all of the above- 
Other advantages and benefits of the invention will mentioned image generation systems. The image data from 
become apparent to those skilled in the art upon a reading these systems are segregated and stored into a memory for 
and understanding of the preferred embodiments. 3S reconstruction and manipulation. Those skilled in the art will 

THMFF DFSCRTPTTON OF THF DRAWINGS appreciated that while the illustration shows medical image 

DESCRIPflON OF YHh DRAWINGS data collectcd mto a common memory 40, the image data 

The invention may take physical form in certain parts and alternatively can reside within plural central, or regional 

arrangements of parts, and in various steps and arrangements memories or memories physically associated with each 

of steps. The drawings are only for purposes of illustrating 40 individual imaging device and mutually accessible, for 

a preferred embodiments and is not to be construed as example, over a network. 

limiting the invention. The COM-architecture layer B includes a plurality of 

FIG. 1 is an overview block diagram of a medical imaging components and interfaces constructed in a component 

system for surgical tool planning/navigation based on com- ob J ect modeling (COM) architecture but written in any 

ponent object modeling architecture; 45 compatible software programming language. For ease of 

HG. 2 is a more detailed diagram of the system of FIG. j5? x f d °° ° f components and interfaces of the 

COM-architecture layer B are grouped in a generic recon- 

' sanction block 42. The graphical user interface layer A is 

FIG. 3 is an exemplary illustration of a medical imaging providcd to man i pu late individual processes within the 

display layout providing communication between compo- 5Q image reconstmction block 42 . For example, a physician 44 

nents in accordance with the present invention; selccts a particular slice a previously acquired^and 

HG. 4 illustrates an exemplary system to register images stored computerized tomo^ap^y^imagejyia the interface 10. 

between real time patient data and previously obtained The user interface 10 selects the appropriate process'from 

image data; and, the plurality of "compoinents in the image reconstruction 

FIG. 5 illustrates an exemplary relationship between a 55 processor~42-for display- 46. r Additionally, surgical-tools or 

graphical user interface and corresponding image displays. in-suite imaging :: systcms^48_(f or example,- spectroscopy 

ncTATT cn nrcPDiDTtnM nc tuc systems) also mtaf acedjhrough compbnenfobject model- 

UbJAlLbU DfcbCKUMION Ob IHfc mg architecture havejheir resultanrimages registered and 

PREFERRED EMBODIMENTS overlaid with user selected images, enabling 4he$hysician to 

FIG. 1 shows an exemplary 2D medical image processing 60 compare real time images-with previously acqinred images 

and 3D,-4D high speed medical image display using control to obtain surgicalrprobe location feedback during lan image 

basea* COM-architecture for an NT operating system. The guided surgery (IGS) procedure^ 

COM based layout can be understood as consisting of three Referring now to FIG. 3, the interface 10 includes a 

layers: the Graphical User Interface layer A, the COM- plurality of processes or algorithms 50 each performing a 
architecture layer B, and the server layer C. The GU1J65 variety of preprogrammed procedures, such as generating a 

interface layer A includes a graphical user interface 10 screen with a 3D rendering and transverse_sagittal and 

which an operatorus^to-mteract wim;image:dala-using a coronal slices mrougb~V~selectelf ? p^ 
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object. When an individual process 50A, for example, other notice of sub-task completion, for example by receipt 

reaches a point on a local instruction set or task requiring of pointers toward a storage location, the transform process 

interaction with another component, a determination of the 80 proceeds with its task as programmed. 

availability of the other component is made. To continue the Referring now to FIG. 5, an exemplary version of the 

previous example, process SOAcan be thought of as request- 5 graphical user interface 10 controls the image displayed on 

ing a 3D^ rendering to rotate. This rotation process, if the display^). As illustrated, buttons,-icpns or other inter- 

availableTcajTreside for example in the COM-architecture face tools are-posi|toned-for„ manipulation by a user to 

layer B which also includes a plurality of individual com- control images on display 46: For example; a button 90 is 

ponents 52A, 52B, 52C. These components 52A, 52B, 52C programmed to toggle or select certain images at a location 

are further comprised of a number of processes 54, for 10 92 in &e in»g c 46. Those skilled in-the art will 

example, potation, coronal slice generation, placement of appreciate a ^similar association can be achieved through 

multiple slices on a single screen, and the like. Moreover the other buttons v in_the interface. Additional flexibility can be 

processes 50 coitained in the interface lO^an represent, for programmed into the interface aUowing ar^operator to rotate 

example, individual display commands such as selecting a sheeted miage, ^ sources, and 

different image^iews^mampulaUng certain image views, 15 P 6 ^™ ot K r ( dispUy^combmati^or manipulations. 

and/or arranging various image views on a display screen. ™ e mvention has been described with reference to the 

Thus, in response to the process 50Areaching apoint in the P rcferr f d ™ b °*™™*- Modifications and alterations will 

algorithm requiring action by a separate object the rota- occur * ^ "f° D V* adin f f md ? f the 

*■ 1 *a ,u *r-iA- a* - a precedmg detailed description. It is intended that the lnven- 

bonal process 54 within the component 52 A is aeterrmned. ^ be * M ^ guch modifications ^ 

T^e miuatmg process SOAthen selects a handle or _ identifier 20 insofar as mey ^ me mpc of the 

60 for the rotation process 54A which is capable of per- a p peD ded claims or the equivalents thereof. 

forming the task. Having thus described the preferred embodiment, the 

Those skilled in the art will appreciate that individual invention is now claimed to be: 

handles 60 areicatalo^ed in a registry table as^objects within 1. A method of displaying medical images, the method 

certain processes are created orrinstanliated. Details of the 25 including: 

registry table interacting with individual objects are known responsive to receipt of a request for manipulation of 
to those skilled in the art. Moreover, artisans appreciate that image data representative of a region of interest, deter- 
within components 52 A, 52B, 52C, individual processes 54 mining a component object within a component object 
also allocate tasks among various other processes, a property model architecture, which component object is config- 
known as Containment. Indeed providing a COM- 30 ured to at least partially accomplish the request; 
architecture for the various components 52A, 52B, 52C in selecting a handle for an interface to the determined 
the architecture layer B provides other handles 62 allowing component object; and 
task sharing between processes within different components processing the request via the handle. 
52A, 52B, 52C. This property is known as Aggregation. In 2 . The method as set forth in claim 1, further including 
other words, the property known as Aggregation enables use 35 displaying the processed image data, 
of one piece of code in multiple COM processes having a 3 ^ me th 0 d as set forth in claim 1, further including: 



common handle 60. 



receiving first image data from at least one medical 



Referring now to FIG. 4, the property of COM Contain- imaging device selected from the set of a magnetic 

ment is illustrated in a registration process between a resonance device, a computed tomography device, a 

previously obtained image and a real time image. Sometime 40 nuclear ^g^g device and an ultrasound device; and 
prior to commencement of a medical procedure, an area of stori ^ fiKt ■ data in a me 
interest ^imaged in a medical imaging device such as a CT 4 ^ method ^ ^ fortn m daim 3j wherein me method 
or other body scanner 70. Tins image data is recor^ructed mchldes lannin a eveot b rformm 

conventionally and stored into an image data memory 72 for ^ mcludin ma nipulaUng a user interface to selectively 

later evaluation and/or use. Later, for example within the 45 OTntrolimage da ta displayed. 

surgical suite during the procedure, additional patient imag- 5 ^ method M ^ forlh m clajm 3> far|her mcludin 

mg information is collected by other imagine apparatus such . . , . , . , 

v 1 -a t^- ■ . . , 0 rr 4 . „ receiving second image data; and 

as a Y-probe 74. This image data is also conventionally . . , . ' . 

processed and stored in a patient information file 76. A * to ™8 "Mood ™«ge data m a memory, 

transform component 78 resides within, or is accessible to, so t 6 - ™? me * od ? et forth in claim ?> whercm ^ method 
the user interface 10. (FIGS. 1, 2). Again, the benefits of a mdudcs Performing a surgical event by performing 

COM-architecture enable various processes to perform spe- tasks m rn^nipulaUng a user interface to selectively 
cific tasks. In the illustrated example, a transform process 80 control display of first image data and second image data, 
is invoked to register or align the current and reference , T ^ metnod ™ xi ^ } n claun X > wherem me 

images from the image data memory 72 and the patient 55 determining a process step mcludes: 
information file 76 prior to display. The transform process is referencing a registry to detect registration of a process 
accessed through its handle 80A. Within the exemplary ada P ted t0 accomplish the request; and 

transform process a selected view is required from the noting the handle associated with the process, 
patient information 76. The transform process 80 formats a 8. The method as set forth in claim 1, wherein the 

request performable by a patient data process 82. The patient 60 component object is determined from a plurality of compo- 
data process 82 is accessed through its handle 82A nent objects within the component object model 
Additionally, the transform process 80 in this example, also architecture, the processing of the request including: 
must manipulate certain previously acquired image data 72. selecting a second handle for an interface to a second 
Thus, the image data process 84 receives the task generated component object; and 

by the transform process 80 through its image data handle 65 formatting a second request directed to the second com- 
84A. Continuing with the example, when the transform ponent object using the second handle to perform at 

process 80 either receives the desired information back, or least a portion of the processing. 
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9. A diagnostic imaging system comprising: 

an image memory system for storing a ptorditvy o a fgglume 
digital image representations reconstruct from diag- 
nostic data generated by a plurality diagnostic imagers; 

a user interface through which a user selects among image 5 
display formats; 

a component object modeling (COM) architecture layer 
which includes image processing components for 
retrieving portions of the digital image representations 
and processing the retrieved portions into the selected 10 
display format 

10. The diagnostic imaging system as set forth in claim 9 
wherein: 

the user interface includes a component ID registry table 
which correlates user selected display formats with IDs 
of selected components for retrieving and processing 
the portions of the image representations. 

11. The diagnostic imaging system as set forth in claim 10 
wherein: 

20 

at least some of the components identify other compo- 
nents which they access and cooperate with to process 
the portion of the image representations. 

12. An image guided surgical system for use by a surgeon 

in performing an image-assisted surgical procedure on a ^ 
surgical area of a subject, the surgical system including: 
a real time surgical imaging system operated by the 

surgeon to acquire real-time images of the surgical 

area; 

a database of diagnostic images of the surgical area 30 
acquired prior to the surgical procedure, the diagnostic 
images including images in a format different from that 
of the real-time images; 

a component object model architecture software layer for 
performing image processing, the component object 35 
model architecture software layer containing a plurality 
of component objects, each component object being 
configured to perform a selected image processing task 
and having at least one interface accessible by a handle; 
and 40 

a graphical user interface that receives the real-time 
images and performs image display processing includ- 
ing at least registering a real-time image with at least 
one diagnostic image, the graphical user interface del- 
egating at least some of the image display processing to 45 
one or more selected component objects of the com- 
ponent object model architecture software layer by 
interfacing with the selected component objects via 
handles of the selected component objects. 

13. The surgical system as set forth in claim 12, wherein 50 
the database of diagnostic images includes diagnostic 
images acquired using at least two different imaging modali- 
ties. 

14. The surgical system as set forth in claim 12, wherein 
the database of diagnostic images includes diagnostic 55 
images acquired using at least two different imaging modali- 
ties selected from a group consisting of: magnetic resonance 
(MR) imaging, computed tomography (CT) imaging, x-ray 
imaging, fluoroscopic imaging, positron emission tomogra- 
phy (PET) imaging, single photon emission computed 60 
tomography (SPECT) imaging, nuclear imaging, and ultra- 
sound imaging. 

15. The surgical system as set forth in claim 12, wherein 
the surgical probe includes an endoscope. 

16. The surgical system as set forth in claim 12, wherein 65 
the component objects of the component object model 
architecture software layer include a component object 



,453 Bl 

8 

selected from a group consisting of: a 3D rendering rotation 
component object, an image registration component object, 
an image overlay component object, an image view selec- 
tion component object, and an image view display arrange- 
ment component object. 

17. The surgical system as set forth in claim 12, wherein 
the component objects of the component object model 
architecture software layer include: 

an aggregation component object defined by aggregation 
of at least two other component objects. 

18. The surgical system as set forth in claim 12, wherein 
the component objects of the component object model 
architecture software layer include: 

a containment component object that contains at least one 
other component object 

19. A image-guided surgical method including: 
acquiring a real-time surgical image during a surgical 

procedure; 

selecting a display format that includes displaying the 
real-time surgical image and at least one stored diag- 
nostic image; 

interfacing with a component object modeling architec- 
ture software layer using handles to selected compo- 
nent objects of said software layer to compute display 
data; and 

generating a display based on the display data. 

20. The surgical method set forth in claim 19, wherein the 
interfacing includes: 

aggregating a plurality of component objects to define an 

aggregate component object; and 
interfacing with the aggregate component object using a 

handle of the aggregate component object. 

21. The surgical method set forth in claim 19, further 
including: 

allocating at least some of the computing of the display 
data to a component object contained by one of the 
selected component objects. 

22. An image guided surgical system for use in perform- 
ing an image-assisted surgical procedure, the surgical sys- 
tem including: 

a real time imager that provides real-time images in a first 
format of a surgical area; 

a database of diagnostic images of the surgical area 
acquired prior to the surgical procedure using one of at 
least two selectable imaging modalities, the diagnostic 
images including images in a second format different 
from the first format of the real-time images; 

a component object model architecture software layer for 
performing image processing, the component object 
model architecture software layer including a plurality 
of component objects, each component object being 
configured to perform a selected image processing task 
and having at least one interface accessible by a handle, 
the component object model architecture supporting at 
least component aggregation and component contain- 
ment; and 

a graphical user interface that receives the real-time 
images and performs image display processing includ- 
ing at least combining a real-time image with at least 
one diagnostic image, the graphical user interface del- 
egating at least some of the image display processing to 
one or more selected component objects of the com 
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ponent object model architecture software layer by 
interfacing with the selected component objects via 
handles of the selected component objects. 
23. The image guided surgical system as set forth in claim 

22, wherein the one or more selected component objects 5 

include: 

an image processing component configured to process 
diagnostic images from any of the at least two select- 
able imaging modalities, the image processing compo- 
nent object including a first imaging modality compo- 
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nent object configured to process images from a first of 
the at least two selectable imaging modalities and a 
second imaging modality component object configured 
to process images from a second of the at least two 
selectable imaging modalities, the first and second 
imaging modality component objects combining to 
contribute to the image processing component object 
by one of aggregation and containment. 

***** 
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